Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tell mysql to ignore the sort index for search queries #32074

Merged
merged 1 commit into from
Apr 25, 2022
Merged

tell mysql to ignore the sort index for search queries #32074

merged 1 commit into from
Apr 25, 2022

Conversation

icewind1991
Copy link
Member

Cleaner version of #29300 (ab)using mysql's inability to use an index for sorting by function.

mysql really likes to pick an index for sorting if it can't fully satisfy the where
filter with an index, since search queries pretty much never are fully filtered by index
mysql often picks an index for sorting instead of the much more useful index for filtering.

@icewind1991 icewind1991 added this to the Nextcloud 25 milestone Apr 22, 2022
@icewind1991 icewind1991 requested review from a team, PVince81, skjnldsv and come-nc and removed request for a team April 22, 2022 11:33
@icewind1991 icewind1991 mentioned this pull request Apr 22, 2022
Closed
@icewind1991
tell mysql to ignore the sort index for search queries
a4e120c 
Signed-off-by: Robin Appelman <robin@icewind.nl>
CarlSchwan
CarlSchwan approved these changes Apr 22, 2022
View reviewed changes
Copy link
Member

@CarlSchwan CarlSchwan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ugly but makes sense :(

@icewind1991 icewind1991 merged commit f7413b9 into master Apr 25, 2022
@icewind1991 icewind1991 deleted the mysql-search-ignore-index-2 branch April 25, 2022 11:23
@icewind1991
Copy link
Member Author

/backport to stable24

@icewind1991
Copy link
Member Author

/backport to stable23

@icewind1991
Copy link
Member Author

/backport to stable22

This was referenced Apr 25, 2022
Merged
Merged
Merged
@bdrewery
Copy link

bdrewery commented Sep 16, 2022 •
edited
Loading

This kills performance for file searches (likes /apps/photos [WITH PREVIEWS EXISTING]) on MariaDB 10.5.

root@localhost [nextcloud]> EXPLAIN SELECT `file`.`fileid`, `storage`, `path`, `path_hash`, `file`.`parent`, `name`, `mimetype`, `mimepart`, `size`, `mtime`, `storage_mtime`, `encrypted`, `etag`, `permissions`, `checksum`, `metadata_etag`, `creation_time`, `upload_time`, `unencrypted_size` FROM `oc_filecache` `file` LEFT JOIN `oc_filecache_extended` `fe` ON `file`.`fileid` = `fe`.`fileid` WHERE ((`mimetype` = 17) OR (`mimetype` = 7) OR (`mimetype` = 22) OR (`mimetype` = 12) OR (`mimetype` = 23)) AND (((`storage` = 3) AND ((`path` = 'files') OR (`path` LIKE 'files/%'))) OR ((`storage` = 1) AND ((`path_hash` = 'e46cc72327dfc3ccaf32f0a167e6c6d1') OR (`path` LIKE '__groupfolders/1/%'))) OR ((`storage` = 7) AND ((`path` = 'files/Shared') OR (`path` LIKE 'files/Shared/%'))) OR ((`storage` = 7) AND ((`path` = 'files/Shared Family') OR (`path` LIKE 'files/Shared Family/%')))) ORDER BY `mtime` + '0' desc LIMIT 40;
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+------------------------+---------+-----------------------+--------+----------------------------------------------------+
| id   | select_type | table | type   | possible_keys                                                                                                        | key                    | key_len | ref                   | rows   | Extra                                              |
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+------------------------+---------+-----------------------+--------+----------------------------------------------------+
|    1 | SIMPLE      | file  | range  | fs_storage_path_hash,fs_storage_mimetype,fs_storage_mimepart,fs_storage_size,fs_storage_path_prefix,fs_path_mimetype | fs_storage_path_prefix | 267     | NULL                  | 522013 | Using index condition; Using where; Using filesort |
|    1 | SIMPLE      | fe    | eq_ref | PRIMARY                                                                                                              | PRIMARY                | 8       | nextcloud.file.fileid | 1      | Using where                                        |
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+------------------------+---------+-----------------------+--------+----------------------------------------------------+
2 rows in set (0.005 sec)

[run query]
40 rows in set (8.793 sec)

Without the + '0' added on it doesn't scan through 500k rows in a temp table.

root@localhost [nextcloud]> EXPLAIN SELECT `file`.`fileid`, `storage`, `path`, `path_hash`, `file`.`parent`, `name`, `mimetype`, `mimepart`, `size`, `mtime`, `storage_mtime`, `encrypted`, `etag`, `permissions`, `checksum`, `metadata_etag`, `creation_time`, `upload_time`, `unencrypted_size` FROM `oc_filecache` `file` LEFT JOIN `oc_filecache_extended` `fe` ON `file`.`fileid` = `fe`.`fileid` WHERE ((`mimetype` = 17) OR (`mimetype` = 7) OR (`mimetype` = 22) OR (`mimetype` = 12) OR (`mimetype` = 23)) AND (((
`storage` = 3) AND ((`path` = 'files') OR (`path` LIKE 'files/%'))) OR ((`storage` = 1) AND ((`path_hash` = 'e46cc72327dfc3ccaf32f0a167e6c6d1') OR (`path` LIKE '__groupfolders/1/%'))) OR ((`storage` = 7) AND ((`path` = 'files/Shared') OR (`path` LIKE 'files/Shared/%'))) OR ((`storage` = 7) AND ((`path` = 'files/Shared Family') OR (`path` LIKE 'files/Shared Family/%')))) ORDER BY `mtime` desc LIMIT 40;
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+----------+---------+-----------------------+------+-------------+
| id   | select_type | table | type   | possible_keys                                                                                                        | key      | key_len | ref                   | rows | Extra       |
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+----------+---------+-----------------------+------+-------------+
|    1 | SIMPLE      | file  | index  | fs_storage_path_hash,fs_storage_mimetype,fs_storage_mimepart,fs_storage_size,fs_storage_path_prefix,fs_path_mimetype | fs_mtime | 8       | NULL                  | 69   | Using where |
|    1 | SIMPLE      | fe    | eq_ref | PRIMARY                                                                                                              | PRIMARY  | 8       | nextcloud.file.fileid | 1    | Using where |
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+----------+---------+-----------------------+------+-------------+
2 rows in set (0.005 sec)

[flush cache and run query without + '0']
40 rows in set (1.168 sec)

Reverting the change allows my /apps/photos to be useable rather than 10 second hangs when scrolling.

@icewind1991
Copy link
Member Author

From what I've seen in large instances, the fs_storage_path_prefix is usually a lot more significant in speedups than the fs_mtime index. Using the mtime index is only really useful if most of the (recent) files belong to the user performing the search, and if the there are significantly more search results than the LIMIT, else it just ends up scanning most of the table sorted by mtime anyway.

@pjft pjft mentioned this pull request May 14, 2023
Open
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skjnldsv skjnldsv skjnldsv approved these changes

@CarlSchwan CarlSchwan CarlSchwan approved these changes

@PVince81 PVince81 Awaiting requested review from PVince81

@come-nc come-nc Awaiting requested review from come-nc

Assignees
No one assigned
Labels
3. to review Waiting for reviews feature: search performance 🚀
Projects
None yet
Milestone
Nextcloud 25
Development

Successfully merging this pull request may close these issues.
4 participants
@icewind1991 @bdrewery @skjnldsv @CarlSchwan